fix(guard): close the two live holes in obligations (c) and (f) - #408
Merged
Conversation
Batch 2 of the multi-persona review of REQ-GUARD-GATE-EVIDENCE-002. Both fixes address a defect that was LIVE on main, and in both cases the previous fix had closed the adjacent question. (f) #385 defect (b): the sorry pattern was as narrow as the exemption The first revision fixed the self-service `-- TODO` exemption (defect a) and left the pattern untouched, so `:= by sorry`, `:= sorry`, `admit` and `axiom cheat : 7 = 8` all reported zero — the four forms the issue named explicitly. An author could assert a falsehood and the floor never moved. Worse, the self-test contained a case ASSERTING that a non-bare occurrence does not count, which made the hole look deliberate. Detection now runs over comment-stripped text (nested `/- -/` and `--`), which is what allows word-bounded patterns instead of line anchors: prose mentioning sorry still does not count, and every real form does. `sorryAx` is counted separately since `\bsorry\b` cannot see inside it. MEASURED before widening: the count is 12 either way. The tree contains no inline sorry, no admit, no axiom, no sorryAx today, so this is a pure tightening with no floor change and no latent red. Output is now broken out by kind, so a shift from `sorry` to `axiom` at a constant total is visible. (c) #404: the filter gate parsed `-p` and then ignored it `extract_filter` extracted the package; the vacuity test matched against a flat workspace-wide inventory. So the gate asked "does this substring occur anywhere?" while run_verification.py runs "does it occur in package P?" — strictly weaker, and #388's own executed proof case survived it. `cargo test -p spar-wasm -- topology` matches three tests, all in spar-solver and spar-network. The inventory is now collected per package and each filter judged against its own. A package that cannot be inventoried is CANNOT-JUDGE (exit 2), not a pass. `--inventory-json` now requires an object; a flat array is rejected because a flat list cannot express which package owns a test, which is the defect itself. TEST-STPA-SVG-TOPOLOGY is demoted to `proposed` rather than repointed. Its description claims eight tests covering labels and connection endpoints; spar-wasm's nearest candidates are the four in `graph::tests`. The described evidence cannot be located, and picking a filter that merely selects something would assert a relevance nobody has checked. Both suites mutation-tested, all mutants caught. Two needed the same fix as check_mutants_report.py — pinning the DIAGNOSIS, not just the exit code — because distinct broken inputs reach the same exit code by different routes, so removing a guard left the suite green. That includes the exact mutant the adversarial reviewer used to falsify the old mutation-testing claim: `filt` -> `filt.split('::')[-1]`, which no case could see because none had a `::` filter while 16 of 60 real ones do. Closes #404. Refs #385 (defect b). REQ-GUARD-GATE-EVIDENCE-002 (c), (f). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rivet verification gate✅ 20/20 passed
Filter: Failed artifacts(none) Updated automatically by |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Found by the per-package inventory the previous commit introduced — its
first contact with a real cargo workspace, and it failed closed as designed.
`crates/spar-cli/Cargo.toml` declares `name = "spar"`. There is no package
called `spar-cli` anywhere in the workspace, and four verification steps
said `cargo test -p spar-cli`:
TEST-CLI cargo test -p spar-cli
VAL-VERIFY-001 cargo test -p spar-cli -- verify
VAL-DIFF-001 cargo test -p spar-cli -- diff
VAL-MCP-001 cargo test -p spar-cli -- mcp
cargo answers `error: package ID specification 'spar-cli' did not match any
packages`, so every one of those steps has always failed when actually run.
The old flat inventory never noticed because it never passed `-p` to cargo
at all — the same discarded-package defect as #404, showing up as a broken
reference rather than a mis-scoped one.
Rewritten to `-p spar`. The filters have candidates under crates/spar-cli/
(verify 24, diff 2, mcp 2), and three of the four are `proposed` in any case.
Also: a package cargo cannot resolve is now a finding ABOUT those artifacts
rather than a traceback. `collect_inventory` distinguishes "did not match any
packages" (omit it, let check() name each affected artifact and exit 2) from
any other cargo failure (still fatal — a broken build invalidates every
verdict, not one artifact's). A traceback names only the first offender;
this names all of them.
19 self-test cases, including one asserting the artifact and package are
named rather than merely exited on.
Refs #404. REQ-GUARD-GATE-EVIDENCE-002 (c).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… tests be an answer Two faults in the per-package collection, both mine, both found by CI on real data. 1. It inventoried packages no filter asks about `main` gathered every package named by any step, including whole-package steps (`cargo test -p X` with no `--`). Those select everything and can never be vacuous, so the inventory is work nothing consumes — and it made the gate FATAL on `spar-verify-macros`, a proc-macro crate with no tests that no filter mentions. Now only packages of steps that actually carry a filter are listed: 9 instead of 10, with eleven whole-package-only crates (spar-annex, spar-dbc, spar-hir, spar-insight, spar-mcp, spar-mermaid, spar-parser, spar-solver, spar-transform, spar-variants, spar-verify-macros) no longer built for nothing. 2. Zero tests was treated as a broken scan An empty list from a SUCCESSFUL cargo run is a truthful answer, not a failure — some packages genuinely have no tests. Keeping the empty set is also the more useful behaviour: a filter on such a package selects nothing, which is a real finding, and reporting it as one beats dying. A broken scan is still caught twice — a non-zero cargo exit stays fatal, and check() exits 2 if the union across all packages is empty. 21 self-test cases. The two new ones pin both behaviours: a whole-package step passes with its package absent from the inventory entirely, and a filter against a package with an empty test set is reported vacuous with the package named. Refs #404. REQ-GUARD-GATE-EVIDENCE-002 (c). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch 2 of the multi-persona review of REQ-GUARD-GATE-EVIDENCE-002. Closes #404,
refs #385 (defect b). Follows #407.
Both defects were live on main, and in both the earlier fix had closed the
adjacent question rather than the reported one.
(f) #385 defect (b): the pattern was as narrow as the exemption it replaced
The first revision fixed the self-service
-- TODOexemption — defect (a) — andleft
_SORRYas the old gate's regex verbatim. So the four forms #385 namedexplicitly all reported zero:
sorryalone on its linetheorem cheat : 1 = 2 := by sorrytheorem cheat : 1 = 2 := sorryadmitaxiom cheat : 7 = 8An author could assert a falsehood as an axiom and the declared floor never
moved. Worse, the self-test contained a case asserting that a non-bare
occurrence does not count — pinning the hole shut rather than finding it, which
makes a gap look deliberate. That case is gone; each form now has one requiring
it to count.
Detection runs over comment-stripped text (nested
/- -/plus--), which iswhat allows word-bounded patterns instead of line anchors: prose mentioning
sorrystill does not count, and cases pin that too.MEASURED before widening — the count is 12 either way. The tree contains no
inline
sorry, noadmit, noaxiomand nosorryAxtoday, so this is a puretightening: no floor change, no latent red, only the hatches closed. Output is
now broken out by kind, so a shift from
sorrytoaxiomat a constant totalbecomes visible.
(c) #404: the gate parsed
-pand then ignored itextract_filterextracted the package at line 165; the vacuity test at line 212matched against a flat workspace-wide inventory. The gate asked "does this
substring occur anywhere?" while
run_verification.pyruns "does it occur inpackage P?" — strictly weaker, and #388's own executed proof case survived
it:
cargo test -p spar-wasm -- topologymatches three tests, all of them inspar-solver and spar-network.
The inventory is now collected per package (10 listings sharing one build) and
each filter judged against its own. A package that cannot be inventoried is
CANNOT-JUDGE (exit 2), not a pass.
--inventory-jsonnow requires an object —a flat array is rejected, because a flat list cannot express which package owns
a test, which is the defect itself.
TEST-STPA-SVG-TOPOLOGYis demoted toproposed, not repointed. Itdescribes eight tests covering labels and connection endpoints; spar-wasm's
nearest candidates are the four in
graph::tests. The described evidence cannotbe located, and choosing a filter that merely selects something would assert a
relevance nobody has checked. A human should decide whether the eight tests get
written or the claim gets withdrawn.
Evidence
obligations found: 12 (declared floor: 12).adversarial reviewer used to falsify the old blanket claim:
filt→filt.split('::')[-1], which every previous case was blind tobecause none carried a
::while 16 of the 60 real filters do.the exit code. Distinct broken inputs reach the same exit code by different
routes (a missing file raises; a missing key reads as
None; anun-inventoried package and an empty inventory both exit 2), so removing a
guard left the suites green. Defence-in-depth and testability pull against
each other, and the resolution is to assert which branch spoke.
That recurrence is the argument for #405's potency harness: mutations run by
hand found survivors in three of five tools this session, and running them by
hand is precisely the unrepeatable practice #407 withdrew a claim about.
Not in this PR
REQ-GUARD-GATE-EVIDENCE-002staysproposedwith itshuman-scopedtag.🤖 Generated with Claude Code